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Abstract 



This Investigation aimed to develop a new methodology for 
assessing the speededness of right-scored tests such as the Test of 
English as a Foreign Language (TOEFL), A new methodology Is needed 
because traditional procedures of assessing speededness assume that the 
test is scored under formula-scoring Instructions. The approach to the 
problem used In this study capitalizes on the fact that Item Response 
Theory Is used In equating TOEFL. Two Indices were Investigated and 
validated against artificial data and real data In a way that was 
consistent with the ETS criteria for declaring forms speeded. Both 
Indices were able to identify data that contained the speed component. 
The applications of these indices to several forms suggest that speed 
does not play a significant role in the TOEFL as a whole. However, 
Section 2 appeared to be affected by speed in administrations where 
pretest Items are included. This may have been due to the slightly 
shorter time per item allowed when pretest items are Included. It was 
recommended that one of the Indices studied be used on an operational 
basis for a trial period and that the time limit for Section 2 be 
increased when pretest Iteras are present or that the number of Items be 
reduced to achieve the same effect. 



Test Speededness Under Number-Right Scoring: An 
Analysis of the Test of English as a Foreign Language 

Isaac !• Bejar 



A test is speeded when some portion of the test-taking population 
does not have sufficient time to attempt every item in the test within 
the allocated time. The effect of speededness depends on, among other 
things, the scoring procedures used with the test. In particular, in a 
multiple-choice test where the total score is based on the number of 
items answered correctly, one effect of speededness could be that 
students who run out of time answer the remaining items in a more or 
less random fashion. They can hardly be blamed for doing so since it 
would be to their disadvantage to leave items unanswered. Unanswered 
items are scored in the same manner as items answered incorrectly, 
therefore, the expected score on those unanswered items is simply zero. 
By contrast, the axpected score under random responding is 1/4 (for 
four-choice items) of the items answered randomly. 

Clearly, to the extent that a test is speeded and students engage 
in random responding, responses to some items will not depend on the 
students level of knowledge, which of necessity tends to reduce the 
validity and reliability of total scores. Hence, it is Important to 
closely monitor the speededness of a test. In the case of the Test 
of English as a Foreign Language (TOEFL) where multiple forms of the 
test are issued, this requires a fresh analysis for each form. The 
analysis that is currently performed, however, is appropriate only for 
tests that are scored with formula scores. Under typical directions 
for formula-scored tests, a series of unanswered items toward the end 
of the test can be taken as evidence that those items were not 
reached. By analyzing the position and number of unreached items it 
becomes possible to evaluate the extent of a text^s speededness. 

The speededness analysis based on Items not reached is clearly not 
appropriate for tests such as TOEFL on which scores are based on the 
number of right answers (ie., without penalty for wrong responses) 
because, as indicated above it is in the student *s Interest to answer 
all items, even if it requires responding randomly. As a result, we 
would expect very few "unreached" items 5 and those unreached items that 
are found would not necessarily be valid indicators of speededness. 
Unreached items may be more an indication that some students did not 
comprehend the instructions. (The students* booklets emphasize that 
the test Is scored according to number right and that it Is in the 
student's interest to answer all questions.) The present study looks 
at speededness from a different perspective, one which does not rely on 
un^^ "^ched Items. 

^ jrpose of the study . The purpose of this study is to develop 
new speededness indices that are valid when a test score is based on 
the total number of correct responses. The approach taken here is 
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based on Item Response Theory (Lord, 1981) since Item Response Theory 
(IRT) provides the psychometric infrastructure of the TOEFL (Hicks, 
1983) • The next section indicates a "theory" of how students are 
likely to approach a test that is in fact speeded. This theory 
provides the motivation for the various indices that are studied. We 
will evaluate the utility of each of these indices for the TOEFL. 

Assumptions 

The most fundamental assumption ve make when speed is not a 
factor is that students' behavior on the test can be modeled by the 
three parameter logistic model. That model expresses the student's 
behavior as the probability of answering an item correctly given the 
student's ability and the item's psychometric characteristics. The 
item characteristics incorporated by this model are difficulty (denoted 
by _b) and discrimination (denoted by a). A third parameter, jc, takes 
into consideration that in a multiple-choice item even examinees at the 
lowest ability levels have a probability greater than zero of answering 
the item correctly. The model that expresses the probability of 
answering an item correctly is depicted by Equation 1. 

1 - c 

^i^®^ " ^ 1 + expl-1.7a^(e - b^)] 

where the subscript i denote the ith^ item and 9 represents ability 
level. 

The validity of Equation 1 depends crucially on the assumption of 
unidiroensionality . According to this assumption the response to 
a given item only depends on that item's parameters and the students 
ability, and nothing else. This assumption specifically precludes the 
possibili^^y that an item's position in the test affects the proba- 
bility of answering correctly the item. For the most part this model 
provides a good description of student behavior on the TOEFL. Figure I 
plots the performance of a sample of over 10,000 students on one item. 
The solid line shows the estimated item characteristic curve (ICC), 
whereas the squares represent the number of students correctly answer- 
ing that item in that interval of ability. The size of the square is 
proportional to the number of students in that interval. As can be 
seen, for these items the fit of the model is good. 



Insert Figure 1 here 



When speed is a factor, we assume that within a section of the 
test students cycle through the section. As they encounter items they 
are not sure of, they leave them for last. The more able student 
would probably attempt all items in one or two cycles but the less 
able student may require more cycles. Since there is a time limit. 
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the less able student may be left with a few unanswered items when the 
time is called, and at that point proceeds to answer the remaining 
items in a fashion that does not depend entirely on ability. 

It is reasonable to expect that the more difficult items would be 
left for last and would thus be subject to random responses. This 
would have several implications both at the item level and more 
globally, which provides a rationale for the indices that will be 
examined. In particular, to the extent chat examinees behave as 
Indicated above, we may find that the 3~parameter logistic model is 
not an accurate model for low ability students on difficult items. 
Specifically examinees of very low ability, by virtue of their behavior 
on the more difficult items, may actually have a higher probability of 
ansvering the item correctly than examinees of slightly higher ability* 
Figure 2 plots performance on an item that would be consistent with 
this forr. of examinee behavior. Notice that while the solid curve 
Increases monotonically with 6, as required by the model, the data 
shows a distinct lack of monotonlcity. For example, students which 
according to the model have an ability around ~2.0 have a higher chance 
of answering the Item correctly than examinees of somewhat higher 
ability. 



Insert Figure 2 here 



We make a similar assumption for section 1. Although section 1 
is paced, that is, each student is presented each item for a fixed 
period of time and there is no opportunity to recycle through the 
test, for the more difficult items the less able students may not have 
sufficient time to respond to the item and rather than leaving the 
item unanswered chey simply do not respond as a function of their 
ability. 

To summarize, the speededness theory proposed here suggests that 
for items above a c<:.rtaln difficulty level examinees of lowest ability 
actually have a higher probability of answering those items correctly. 
An empirical implication of the theory is the presence of non- 
monotonicity of probability of a correct response as a function of 
ability. Although it is not pretended that the theory describes 
different strategies students may use in going through the test, it is 
one possible explanation for the lack of monotonlcity that are observed 
sometimes with difficult items. 

Approach 

The approach taken in this investigation is to generate data that 
meets our assumptions when speed is present and when it is not. How 
this is done will be described shortly. Then we study two types of 
indices, what may be called an item-level index and a examinee-level 
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Index. Each of these indices is validated, first against artificial 
data, and then with data from several real TOEFL forms. These 
analyses are based on Item Response Theory which, as previously 
indicated, is the psychometric foundation of TOEFL. 

Description of the indices 

Item-based index . The assumptions outlined earlier regarding 
specdedness suggest that a reasonable item-level index is simply to 
compute for each item a discrepancy measure between the estimated item 
characteristic curve, that is Equation 1, and the data. The estimation 
of Equation 1 is a complex process. Fortunately, a program, LOGIST, 
exists for performing the estimation. (Wingersky, Barton and Lord, 
1982). 

A model-free estimate of Equation 1 can be obtained by simply 
grouping examinees in intervals of ability and computing the proportion 
of students within each interval that answer the item correctly. This 
second estimate is not truly model-free since we group the examinees 
based on their estimated ability and the estimation of ability is based 
on the model. Nevertheless, the second estimate would not be 
constrained in the same fashion that the estimate resulting from the 
LOGIST program is. In particular, estimates obtained from LOGIST are 
subject to the constraint that the probability of a correct response 
increases with ability, whereas the "model free" estimate is not 
constrained in that fashion. Therefore, a comparison of the LOGIST 
estimate and the model-free estimate can at least detect items where 
the probability of a correct response does not increase with 9. We are 
especially interested In this kind of discrepancy since, as outlined 
earlier, it is the kind of discrepancy expected when speed is present. 

We will focus our attention on an Index proposed by Yen (1981). 
The Index is generic in the sense that it can be used to detect lack 
of fit at the item level regardless of the source. The index, called 
Q, has been modified slightly by Cook and Douglass (1982). It is 
computed as follows. Examinees are rank ordered In terms of their 
estimated ability and grouped into 15 equally spaced intervals of 6 
from -3.00 to 3.00. If any cell contains fewer than 5 examinees, it 
is collapsed with the neighboring cell closest to 9 = 0. 

The formula for the index is given by Equation 2, 

15 [N (0 - E )]2 
Q = E ^ ^ (2 



where 



E^jd - E^.) 



N is the number of examinees in cell i 
2 ^ 

0 is the observed proportion of examinees in 
^ cell j that answers the ith item correctly 



ii the ptifidictcd proportion of examtnati in 
^ cell j that answer the ith item correctly 
That is, 

^ij " ^^/^j^ ^ ^i^V 

kGj 



where 



P^CSj^) is given by Equation 1. 



An examinee-based index * The above index focuses on the item as 
the unit of analysis • That is, we look for items that may not be 
performing according to ths model. A. complementary approach to 
detecting speededness is to look for examinees that fail to perform as 
predicted by the model. There is a fair amount of research on this 
topic (Levine and Drasgow, 1982, for a review), although no applica- 
tions to the speededness problem have been made. The purpose of this 
index is to detect students who get a spuriously high or low score. In 
the present application our focus is on low ability students who get 
spuriously high scores. By spuriously high I mean that they get a 
score higher than would be predicted if they were responding as the 
model expects them to. 

Clearly, there is a certain amount of circularity in these 
indices for if we find such examinees then the model needs to be 
corrected. For our purposes however, this circularity is reduced 
because in practice we can not continue tc reestimat^^ the mod^r^l 
indefinitely. The model here ii, understood to mean the 3-parameter 
logistic model since that is the »todel used by the program to maintain 
a common scale across forms. 

Just as the Item-level index involves a comparison between the 
prediction of the model and the actual data, the examinee-level index 
also involves a comparison of the prediction of the model and the 
actual data. Perhaps, the most meaningful index is one that compares 
the predicted number of responses correct and the actual number of 
correct responses. However, since the focus of the present investi- 
gation is on speededness the index adopted for investigation is the 
discrepancy between observed and predicted performance on the most 
difficult items. The choice of this index is in line with the 
assumption made earlier that students cycle through the test and leave 
the most difficult items for last. Thus, it is the most difficult 
items that would be vulnerable to speeded testing conditions. 

Computation of index . The computation of the test-level 
index is as follows: 

For each separately timed section of length 1^ reorder the items 
so that they are in increasing order of difficulty, regardless of 
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Item types . (It is often the case that a test contains more 
than one item type and that within an Item type items are ordered 
in difficulty. For the purposes of this analysis we ignored the 
item ijpes and simply ordered the item in increasing order of 
difficulty.) 

Next, we divide the section into an *'easy" and a "hard" part. The 
division point is obviously arbitrary but we will adopt a 
division that is compatible with the standard used at ETS for 
declaring a form as speeded. According to that standard, a 
section is speeded if fewer than BOX of the examinees reach the 
last item on the section and fewer than iOOX complete 75% of the 
section. Thus, we will divide the section into two with the easy 
part containing the 75% easiest items and the hard part containing 
the 25% hardest items. 

Now we compute the expected and observed performance on the hard 
part. Again, the rationale for this is that it is the 
performance on the most difficult items that is vulnerable to 
speed. The expected performance is computed by 

E(e^) - s P^(e^) (A 
i 

where is given by Equation 1, the sum is taken over the 
hard Items and 9^ is the estimated ability based on the easy 
items. In other words, we compute the expected performance 
on the hard items based on the ability estimate computed on 
the easy items. 

The observed performance on the hard items is simply 

0 = E u^ (5 
i 

where u^ is 1 if the ith item is answered correctly and 0 
otherwise, and the sum is taken over the hard items only. 

Evaluation of indices with artificial data 

Since there is no prior information on the indices under study a 
significant portion of this investigation was devoted to the evaluation 
of the IRT indices with artificial data. This approach allows us to 
examine the behavior of the indices when speed is a factor and when it 
is not. Put briefly, the evaluation consists of generating data that 
fits the 3~parameter logistic model but having statistical character- 
istics similar to those found in a typical TOEFL form. The performance 
of the indices with these data will let us know what to expect when in 
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fact speed Is not a factor. We then Introduce a speed effect Into 
these data. Clearly, In order to do this we must make several assump- 
tions that will be outlined below. 

Generation of data . The procedure for generating data according 
to the 3-parameter model requires that we specify the Item parameters 
and the distribution of ability. Since we are concerned with simu- 
lating TOEFL data we will choose item parameters from a typical TOEFL 
form (November 1982). For concreteness , we will assume that the dis- 
tribution of ability is normal- The a, b^, and £ parameters for simu- 
lating data for sections 2 and 3 are found In Appendix A. (Although 
the validation with real data was based on all three sections, the 
validation with simulated data was based only on sections 2 and 3-) 
These are simply the "operational" parameter estimates for the November 
1982 administration. By using these parameters, we Insure that the 
simulated data have the characteristics of dar.a from typical TOEFL 
form. With the item parameters and distribution of ability in hand, 
the procedure to generate the data Is as follows: 

Sample an ability value from a normal distribution with mean 0 
and standard deviation of 1.0 

For each item Compute the probability of a correct 
response by evaluating Equation 1 substituting the 
sampled ability and the parameters for this item. Call 
the obtained probability p. 

Then: Generate a random number from a rectangular 
distribution in the interval 0 to 1.0. Call this 
number p*. 

If p > p*, then the response to this item is correct, 
otherwise it is Incorrect. 

We record the outcome of the simulation in a data 
matrix U, such that u.^ is the jth subject response 
to the l£h item. 

Repeat for each item. 

Repeat for each sampled ability value. 

The outcome of this procedure is a data matrix with as many rows 
as there are ability values and as many columns as there are items* We 
have chosen 3,000 as the sample size. For the simulation of section 2 
we have 38 items, and 58 for the simulation of section 3. We will call 
this matrix for power, since it contains data that fits Equation 1 
and it is uncontaminated by speed. We now need to modify the P matrix 
in such a way that the effect of speed is simulated in a form that is 
consistent with the ETS criteria for speededness. 
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The procedure to Introduce speed In the artificial data consists 
of changing the responses to the 25Z most difficult Iteas of examinees 
in the lover 20Z of the ability distribution. This does not parallel 
the conventional criterion since neither difficulty nor ability come 
into the definition but It does follow the opirlt of the traditional 
criterion. 



Insert Figure 3 here 



Figure 3 shows the structure of an S matrix where the ability 
levels have been ordered from high to low and the items have been 
ordered in difficulty. We assume that responses In all but the 
"southeast" region of this matrix are modeled by the 3-parameter 
logistic model. Responses In that region are subject to speededness 
because they result from fhe interaction of students with the lowest 
ability attempting the most difficuU items. To match the spirit of 
ETS criteria for speededness, that region is defined by the lowest 20Z 
of the ability values and the 25% most difficult items. According to 
our "theory*' of speededness, responses in the "southeast" follow one of 
two models; the usual 3-parameter logistic model and a random response 
one. As ability gets lower and the items get more difficult the proba- 
bility that the usual model applies diminishes while the probability of 
a random responses Increases. Within an interval of ability, the prob- 
ability of a random response increases in a gradual fashion. Figure 3 
illustrates the scheme for a 40-item test. In a 40-1 tern test there 
would be 10 items subject to speed. Then the interval from the 20th 
percentile to -3.0 is divided into 10 subintervals , or in general as 
many subintervals as there are items subject to speed. The width of 
the subintervals is given by 

w - (9* + 3)/ng (6 

where n^ is the number of items subject to speed and 6* is the 
ability value corresponding to the 20th percentile. Let us call the n 
interj/als i, where k goes from 1 to n^ and « k. For examplef 
if e = -2.0, then w = .10. Similarly, let u^ refer to the 25Z most 
difficult Items by j where jj is the least difficult among those 
items and j^^ = 10 refers to roost difficult item. Then the proba- 
bility of a random response is given by 

f (iJi)/n k,l = l..cn^ if r,, < 1.0 

hi--] ^ (7 

^ I 1.0 if r^^ > 1.0 

With these definitions in hand we can now describe the S data 
matrix as follows 
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If the ability level Is above the 20th percentile the 
probability of a correct response is given by the 
3-parameter logistic aodel • 

For ability levels below the 20th percentile correct 
responses to the easiest 75X of the Items are also 
given by the 3-parameter logistic model ♦ 

For ability levels below the 20th percentile correct, 
responses to the 25% most difficult Items are given by 

.25 with probability r^j 

P(9) with probability 1 - r.. 



Results for artificial data 

E xaminee-index . In this section we present results relevant to 
the evaluation of the proposed indices with artificial data. Table 1 
shows the result based on one sample of approximately 2000 subjects 
with normally distributed ability for both section 2 and section 3* 
The first step in the computation of this table ^.s to classify response 
vectors in intervals based on estimated theta for all the items. Next, 
we compute the mean number right and predicted number right score in 
each interval. Finally, the difference between the number rijjht and 
predicted number right at each interval is an index of the effect of 
speed* 

The columns labeled *'P" are for the data under power conditions, 
while the column labeled "S" is for the speeded condition. The 
difference was computed by substracting PNR, the expected nurabec-right 
from the observed number-right, NR» Thus, a negative difference 
indicates that the expected number-right exceeded the observed score. 

We note that as 9 increases the differences become positive and 
increase in magnitude. In other words, regardless of the presence of a 
speed factor the estimate of ability based on the easiest items appears 
to be an underestimate for high levels of ability. This was not 
expected. However, since we are interested in the other extreme of the 
scale, it is not pertinent to our results. 



The effect of the presence of speed can best be examined by 
plotting the differences between observed and expected performance. 



Insert Figure 3 here 



Insert Table 1 here 
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Figure. 4 shows Che index for section 2 data. The curve labeled "S2" Is 
for the data for section 2 with a speed factor* The curve labeled *'P2*' 
Is for the same data without a speed factor. As can be seen "S2" 
bulges towards the low values of ability. This Is an Indication that 
when a speed factor Is present low ability "subjects" get spuriously 
high scores. 



Insert Figure A here 



Figure 5 shows corresponding results for section 3. The curve 
labeled "P3" Is for data simulating section 3; the curve "S3" Is for 
the same data with a speed factor added. We see again that there Is a 
bulging toward the lower ability values. 



Insert Figure 5 here 



The above results are not entirely applicable to real TOEFL data. 
ITie reason for this lies in the way the TOEFL is equated. The 
procedure in current use is called "fixed b's" equating (Hicks, 1983). 
As its name implies, that method is based on fixing the difficulty 
estimate of some of the items to the value obtained using a different 
sample. However, only the difficulty or bi parameter is held fixed; the 
other parameters, £ and a^ are reestimated. As a result, there is an 
opportunity for Improving the fit of the items at equating time. That 
opportunity is missing from the results just presented. 

In order to approximate the effect of the opportunity to adjust 
the parameter estimates, the £ and £ parameters of all the items were 
reestimated and the index recomputed. The results are shown in Table 
2. 



Insert Table 2 here 



The differences between the expected number right and the observed 
number right for sections 2 and 3 are plotted in Figures 6 and 7 
respectively. As can be seen, the effect of speed is still present 
after reestlmatlng the parameters. 



Insert Figures 6 and 7 here 



Item-based index . These results suggest that the index proposed 
here is in fact capable of detecting situations when a speed factor, 
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thus meeting the spirit of ETS criteria* We now would like to examine 
the perfosrmance of the item-based index discussed earlier, Q. 

We first compute the index for each Item for the P and S datasets 
for sections 2 and 3* Our Interest Is to see If the value of the 
index Is higher for the most difficult items when they are subject to 
speed* Rather than present the raw results, a curve was computed 
containing the running average of Q beginning with the most difficult 
Item. Thus, the right-most point in this curve is just the value of Q 
for the most difficult item, the second point Is the average of Q for 
the two most difficult items, the last point In the curve is just the 
average of Q across all the items. These mean values are then plotted 
as a function of the difficulty of the most difficult Item for a given 
point. For example, the first point, which is the value of Q for the 
most difficult item, is plotted against the difficulty of that item; 
the second value is plotted against the difficulty of the second most 
difficult item. 

The results for sections 2 and 3 appear In Figures 8 and 9 
respectively. For section 2 under power conditions the running 
average remains fairly constant although there is a slight tendency 
for it to increase as a function of difficulty. However, the contrast 
with the curve for data under the speed condition is obvious. The 
mean is almost three times as high for the most difficult items, which 
of course are the items subject to speed, and then decreases sharply. 
The pattern Is much the same for section 3: under power conditions the 
running average Is fairly constant but it increases as a function of 
difficulty when speed is present. 



Insert Figures 8 and 9 here 



Conclusions based on the artificial data . Based on the results 
presented for the examinee-based and itera-based Index, It can be 
concluded that both indices are able to Identify the presence of 
speed. We can be reasonably confident that the contrast between the 
Indices under the power and speed condition are due to speed because of 
the nature of the artificial data. Thus, other things being equal. It 
is appropriate to conclude that If both indices behave as they did here 
we may be justified In concluding that a form is speeded. In practice, 
however, there may be other deviations from the model present In the 
data due to factors other that speed, which naturally will make the 
interpretation of these indices less certain. Nevertheless, the above 
rf^sults gives us an Indication of what would happen under idealized 
circumstances and should be valuable In the interpretation of these 
indices when applied to real data. 

Results with real data 

We now turn to the results based on actual TOEFL data* We will 
present results with the two indices for the November 1982 
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administration. Additional results using the Item level index will be 
presented for four recent administrations* 

Results for the examinee-based Index * First we present the 
results of the examinee-based Index. Table 3 contains the results of 
applying the procedure to a spaced sample of approximately 2,000 
examinees from the November 1982 administration. We notice a similar 
increase in the difference between NR and PNR for high values of 
ability. But, as before, our interest lies in the other direction. To 
better see the results. Figure 10 shows the index for sections 1, 2, 
and 3. 



Insert Table 3 here 



Insert Figure 10 here 



Figure 10 suggests that sections 2 and 3 are not affected by 
guessing but sec. 'on 3 appears to show a similar pattern to the 
simulated data with a speeded factor. This sugggests that a speed 
factor mav be present in section 3. To add credence to this 
suggestion, a different spaced sample from the same administration was 
examined. (Only the results for section 2 and 3 are presented because 
section 1 did not converge after spending a considerable amount of 
time). The results appear in Table 6 and a corresponding plot in 
Figure II. 



Insert Table 4 here 



As can be seen a similar pattern emerges: there is no "bulging" 
of the curve at low ability levels for section 2 but thete is for 
section 3. As with the first sample the frequency of the lowest three 
ability intervals is 0 for section 2, which precludes the possibility 
of bulging in that range of ability. 



Insert Figure 11 here 



Results for the item-index with real data . Tables 5, 6, and 7 
display the item parameter estimates, for sections 1, 2, 3 
respectively, for the November 1982 administration. The tables also 
contain the value of the Q index associated with each item. Figure 12 
shows the average Q value as a function of difficulty, computed In the 
same manner as with the artificial data. 
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Insert Tables 5, 6, and 7 here 



Insert Figure 12 here 



For section 1 the index rises slowly from a value of about 30 to 
about 45; for section 2 the index also raises slowly but from 20 to 
about 40; finally for section 3 the index remains fairly constant at 20 
except that it decreases somewhat for high values of difficulty. 
Several observations are in order. The best indication we have of 
what the value of Q should be when the data fits the underlying model 
is from the P data sets. For those datasets the value is approximately 
15 throughout the range of difficulty. It is interesting to note that 
for section 1 with real data the value is approximately 30 and rises 
somewhat as difficulty increases. However, the increase is nowhere in 
magnitude of what it was for the S data sets. Thus, although the fit 
is not as good as it is for the artificial data it seems that the lack 
of fit is not necessarily due to the presence of speed. For example, 
item 14 of Section 3 had an extremely high value of Q. Examination of 
this item indicates that the reason is a shift in the difficulty of the 
item. As stated earlier, in equating, only the £ and £ parameters are 
reestimated. The b^ parameter, difficulty, is held fixed from pretest 
administration to final administration. Such shifts in difficulty are 
reflected in high Q values. 

For sections 2 and 3, the fit at low difficulty values is somewhat 
higher than it is for artificial data that fits the model* For section 
3 the fit does not seem to depend on difficulty but for section 2 there 
is a tendency for lack cf fit to increase with difficulty although not 
at the same raie zz Ic does with artificial data with a speed 
component. 

Further analysis . The results just presented suggest that the 
November 1982 form is not speeded, using as criteria the expectation 
provided by artificial data subject to speededness to an extent 
consistent with ETS conventions. Although Section 3 appeared speeded 
on the examinee-level index, it did not appear speeded on the 
item-level index. Similarly, Section 2 appeared speeded on the 
item-based index, although it did not reach the level found with the 
simulated data, and it did not appear at all on the examinee-based 
index. It became apparent, however, that the computation of the 
examinee-based index is considerably more difficult and expensive- 
Indeed, in one instance the process ran out of computer time. 
Therefore, it appears that for operational purposes the item-based 
index is to be preferred. 

In this section we present additional results for the item-based 
index on additional administrations. Specifically, the item-based 
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Index was applied to all sections of the September, October, November, 
and December 1983 administrations. Tables showing the Item parameters 
and Q Index can be found In Appendix A. The results are summarized in 
Figures 13 through 16. 

An examination of the figures suggests the following: for section 
1 the Item-based Index remains flat as difllculty Increases. However, 
as with the results presented earlier, the general elevation of the 
plot Is higher for Section 1 than for Sections 2 and 3. For Section 3, 
again as before there Is no evidence that speed plays a significant 
role and also as before the general elevation of the plot Is the 
lowest. For Section 2, however, there Is a tendency for the plot to 
Increase with difficulty. For the September administration there Is a 
marked Increase of the plot beyond a difficulty value of 1.0. A 
similar Increase Is noticed for the November administration but not for 
the October administration. 

For the December administration the Index for Section 2 appears 
similar to the plot that we obtained with artificial data In that the 
plot begins to rise at around -.5 and actually reaches the level 
comparable to the artificial data with a speed factor. Thus, It would 
seem that Section 2 for the December administration Is speeded. 
Because of the Interrelationship between ability and Item difficulty In 
the present approach to assessing speedadness, the apparent speededness 
of the December 1983 administration may have arisen because in that 
administration the examinees were of lower ability or the items were 
unusually difficult. Table 8 reports the mean and standard deviations 
for the September 1983 through December 1983 administration on the 
three TOEFL sections' scores as well as the total score. The sample 
sizes on which these statistics are computed are shown as well. As can 
be seen, it does not appear that the December examinees are of lower 
ability. 



Insert Table 8 here 



Next we looked at the mean difficulty of the September to December 
1983 administrations to see if in the December form Section 2 was 
unusually difficult. The results are shown in Table 9. It does not 
appear that Section 2 was harder in the December form than in 
September, October, or November. However, the variability of the 
difficulty and the discrimination present is highest for December. 

Insert Table 9 here 



Summary and Conclusions 

This investigation aimed to develop new methodology for assessing 
the speededness of TOEFL test forms. This methodology is needed 
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because the traditional procedures of assessing speededness assume the 
test is scored under formula-scoring instructions whereas TOEFL is 
scored under number-right scoring instructions. The approach to the 
problem capitalizes on the fact that for TOEFL Item Response Theory is 
the equating methodology used. 

Two complimentary indices of speededness were investigated. Both 
of these indices are based on the assumption that if a test is speeded, 
performance on the most difficult items will not be entirely a function 
of ability. One index contrasts the observed performance on the most 
difficult items in the test against the performance by the model for 
those items. The second index is based on an index of fit of each 

1 tem. 

Both indices were validated by creating data that fit the model 
and then introducing a speed component into the data. This was done 
in a way that was consistent with the ETS criteria for declaring forms 
speeded. Both indices were able to identify data that contained the 
speed component. The indices were then applied to real data from the 
November 1982 TOEFL administration. The results suggest that none of 
the sections is speeded. The application of the examinee-based index 
suggested that Section 3 might be speeded, but the Item-based index did 
not indicate this. For Sections 1 and 2, the examinee-based index did 
not suggest either section was speeded. The item-based index increased 
with item difficulty, which would be consistent with the presence of 
speed, but the increases were not as large as would be expected if 
speed were present to the extent necessary to meet the conventionally- 
inspired criteria for declaring a test speeded. 

Ic became apparent during the investigation that the examinee- 
level index was expensive to compute. Therefore, the item-level index 
was applied to data from four recent administrations. There was no 
indication that Sections 1 or 3 were speeded, but Section 2 of the 
December form tended to appear speeded. The reason for this apparent 
effect was not that the December form was harder or the population less 
proficient. Rather the effect seems to be due to the slightly differ- 
ent time allocation for each section. The time allowed for Section 2 
is 25 minutes for 40 items or .62 minute per item. If there are pre- 
test items, however, the allowance of time per item is 35/60 = .58 
minutes per item. To maintain an identical time allowance, 37.5 
minutes should be allowed for the entire section instead of the current 
35 minutes. Therefore, when there are pretest items in Section 2, 

2 1/2 more minutes should be allowed to maintain identical timing 
constraints. For Section 3 without pretest items, the time allowance 
is 35 for 60 items of .58 minute per item. If there are pretest items, 
the time allowance is 65 minutes for 90 items oi .72 minute per item. 

The November and December administrations were pretest; and it was 
these two administrations, especially December's administration, that 
appeared more speeded according to the item-level index. Based on 
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these results It is tecommended that the Item-based index proposed here 
be implemented by the program, at least on a trial basis, to monitor 
the speededness, or lack of it, of future TOEFL forms. If further 
application of the index continues to suggest that Section 2 may be 
somewhat speeded, it may be advisable to increase the time allocation 
for Section 2 when pretest items are available by 2 1/2 minutes. 
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Figure Description 

1 Empirical and theoretical Item characteristic 

function based on a sample of over 10,000 
examinees for an Item that fits the model well. 

2 Empirical and theoretical Item characteristic 

function based on a sample of over 10,000 
examinees for an item that exhibits lack of 
fit consistent with the presence of speed* 

3 Data matrix for data with a speed factor. 

A Plot of the difference between NR and PNR for 

Section 2 data under power and speed conditions 
before adjusting Item parameter estimates. 

5 Plot of the difference between NR and PNR for 

Section 3 under power and speed conditions 
before adjusting Item parameter estimates. 

6 Plot of the difference between NR and PNR for 

Section 2 data under power and speed conditions 
after adjusting Item parameter estimates. 

7 Plot of the difference between NR and PNR for 

Section 3 under power and speed conditions 
after adjusting Item parameter estimates. 

8 Average Q as a function of difficulty under 

speed and power condition for Section 2 
data. 

9 Average Q as a function of difficulty under 

speed and power condition for Section 3 
data. 

10 Plot of the difference between NR and PNR for 

Sections 1, 2, and 3 based on data from 
approximately 2,000 examinees from the 
November 1982 administration. 

11 Plot of the difference between NR and PNR for 

Sections 1, 2, and 3 based on a replication 
sample of approximately 2,000 examinees from 
the November 1982 administration sample from 
Sections 2 and 3. 
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12 Plot of item-based Index for Scctlona 1, 2, 

and 3* The top panel Is for Section 1, the 
middle panel for Section 2) and the bottom 
panel for Section 3. 

13 Plot of Item-based Index for Section 1 (top panel), 

2 (middle panel), and 3 (bottom panel) for the 
September 1983 administration based on 
approximately 2,000 examinees. 

14 Plot of item-based index for Section 1 (top panel), 

2 (middle panel), and 3 (bottom panel) for the 
October 1983 administration based on 
approximately 2,000 examinees. 

15 Plot of item-based index for Section 1 (top panel), 

2 (middle panel), and 3 (bottom panel) for the 
November 1983 administration based on 
approximately 2,000 examinees. 

16 Plot of item-based index for Section 1 (top panel), 

2 (middle panel), and 3 (bottom panel) for the 
December 1983 administration based on 
approximately 2,000 examinees. 
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Table 1 



NR, PNR and the difference between the two for data sets 
with and without speed for sections 2 and 3 as a function of 
estlinated ability without adjusting item parameters. 



* 


P 2 






P 3 






S 2 






S 3 




P2 


S2 


P3 


S3 




nr 


pnr 


diff 


nr 


pnr 


diff 


nr 


pnr 


diff 


nr 


pnr 


diff 


N 


N 


N 


N 


-2.6 


* 


* 


* 


2.50 


2.97 


-.47 


* 


* 


* 


3.00 


2.96 


.04 


0 


0 


2 


1 


-2.2 


1.50 


1.55 


-.05 


2.67 


3.01 


-.34 


2.50 


1.55 


.95 


4.22 


3.00 


1.22 


2 


2 


9 


9 


-1,8 


1.68 


1.62 


.06 


2.79 


3.07 


-.28 


3.00 


1.97 


1.03 


3.28 


3.08 


.20 


25 


21 


19 


18 


-1.4 


1.24 


1.66 


-.42 


3.50 


3.21 


.29 


2.40 


1.65 


.75 


3.59 


3.21 


.38 


55 


47 


48 


41 


-1.0 


1.75 


1.79 


-.04 


3.30 


3.43 


-.13 


2.07 


1.78 


.29 


3.6'J 


3.42 


.27 


115 


114 


94 


103 


-.6 


2.10 


2.04 


.06 


3.86 


3.83 


.03 


2.28 


2.00 


.28 


3.93 


3.83 


.10 


203 


195 


229 


235 


-.2 


2.33 


2.54 


-.22 


4.64 


4.52 


.12 


2.45 


2.54 


-.08 


4.64 


4.55 


.09 


333 


332 


322 


316 


.2 


3.17 


3.44 


-.27 


5.76 


5.72 


.04 


3.16 


3.27 


-.11 


5.77 


5.72 


.05 


416 


414 


413 


413 


.0 


4.49 


4.67 


-.18 


7.25 


7.38 


-.13 


4.49 


4.63 


-.14 


7.25 


7.38 


-.13 


340 


339 


385 


385 


1.0 


5.88 


6.27 


-.39 


9.03 


9.28 


-.25 


5.87 


6.27 


-.40 


9.03 


9.28 


-.25 


236 


235 


248 


248 


1.4 


7.55 


7.07 


.48 10.70 


11.01 


-.31 


7.55 


7.07 


.48 


10.70 


11.01 


-.31 


111 


111 


137 


137 


1.8 


8.68 


7.68 


1.00 


11.97 


12.40 


-.44 


8.68 


7.68 


1.00 


11.97 


12.40 


-.44 


50 


50 


29 


29 


2.2 


9.58 


7.84 


1.74 


13.17 


12.62 


.55 


9.55 


7.92 


1.62 


13.17 


12.62 


.55 


12 


11 


23 


23 


2.6 


10.00 


8.23 


1.77 


13.87 


13.61 


.26 


10.00 


8.23 


1.77 


13.87 


13.61 


.26 


5 


5 


15 


15 


3.0 


10.00 


8.87 


1.13 


14.60 


13.36 


1.24 


10.00 


8.87 


i.:3 


14.60 


13.36 


1.24 


2 


2 


5 


5 
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Table 2 



NR, PNR and the differnce between the two for data sees 
with and without speed for setions 2 and 3 as a function of 
estiJiiated ability after adjusting item parameters* 



* 


P 


2 






P 3 






S 2 






S 3 




P2 


S2 


P3 


S3 


TH 


nr 


pnr 


diff 


nr 


pnr 


diff 


nr 


pnr 


diff 


nr 


pnr 


diff 


N 


N 


N 


N 


-2.e. 


* 


* 


* 


3.25 


• 2.98 


.27 


* 


* 


* 


3.50 


3.05 


.45 


0 


0 


f 

H 


2 


-2,2 


1.67 


2.19 


-.52 


2.60 


3.02 


-.42 


2.25 


1.65 


.60 


4.00 


3.09 


.91 


6 


4 


5 


8 


-1.8 


1.70 


1.48 


.22 


2.86 


3.08 


-.22 


3.18 


1.69 


1.49 


3.13 


3.14 


-.01 


20 


17 


21 


23 


-1.4 


1.33 


1.74 


-.41 


3.36 


3.23 


.12 


2.43 


1.77 


.67 


3.48 


3.28 


.20 


48 


44 


45 


50 


-1.0 


1.72 


1.77 


-.05 


3.41 


3.45 


-.04 


2.04 


1.90 


.15 


3.66 


3.49 


.18 


109 


117 


95 


109 


-.6 


2.11 


1.90 


.21 


3.90 


3.86 


.05 


2.39 


2.10 


.28 


3.91 


3.89 


.02 


198 


199 


223 


235 


-.2 


2.31 


2.40 


-.09 


4.63 


4.52 


.11 


2.46 


2.60 


-.14 


4.61 


4.57 


.04 


327 


341 


317 


327 


.2 


3.16 


3.19 


-.03 


5.78 


5.70 


.08 


3.17 


3.38 


-.21 


5.78 


5.74 


.04 


414 


415 


419 


424 


.6 


4.47 


4.52 


-.05 


7.25 


7.37 


-.12 


4.54 


4.71 


-.17 


7.25 


7.41 


-.17 


345 


348 


374 


378 


1.0 


5.86 


6.15 


-.29 


8.98 


9.17 


-.19 


5.96 


6.31 


-.36 


9.01 


9.27 


-.26 


225 


224 


242 


246 


1.4 


7.48 


7.00 


.48 


10.59 


10.91 


-.31 


7.55 


7.13 


.42 


10.66 


11.05 


-.39 


121 


106 


138 


136 


1.8 


8.65 


7.73 


.93 


11.95 


12.20 


-.25 


8.53 


7.67 


.86 


12.03 


12.46 


-.43 


46 


49 


38 


35 


2.2 


9.38 


7.68 


1.71 


13.21 


12.77 


.44 


9.50 


7.74 


1.76 


13.35 


12.85 


.50 


13 


10 


24 


23 


2.6 


10.00 


8.27 


1.73 


13.92 


13.90 


.01 


10.00 


8.46 


1.54 


13.92 


13.99 


-.07 


4 


6 


12 


12 


3.0 


10.00 


8.84 


1.16 


14.40 


13.38 


1.02 


* 


* 


* 


14.50 


13.74 


.76 


2 


0 


5 


5 
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Table 3 



NR, PNR and difference for real data set from the November 

1982 administration 



TH 


Section 


1 




Section 


2 


Section 


3 


Section 


3 


nr 


pnr 


diff 


nr 


pnr 


diff 


nr 


pnr 


diff 


N 


N 


N 


-2.6 


* 


* 


* 


* 


* 


* 


* 


* 


* 


0 


0 


0 


-2.2 


3.22 


2.89 


.32 


* 


* 


* 


4.14 


3.01 


1.13 


23 


0 


7 


-1.8 


3.17 


2.97 


.20 




* 


* 


3.27 


3.08 


.18 


36 


0 


15 


-1.4 


3.20 


3.15 


.05 


1.83 


1.65 


.19 


3.20 


3.20 


.00 


83 


65 


45 


-1.0 


3.26 


3.46 


-.20 


1.71 


1.79 


-.08 


3.70 


3.40 


.29 


203 


118 


102 


-.6 


3.92 


4.03 


-.11 


2.U5 


2.02 


.03 


3.89 


3.82 


.08 


279 


198 


194 


-.2 


4.99 


4.90 


.09 


2.36 


2.53 


-.17 


4.64 


4.54 


.10 


391 


332 


360 


.2 


6.45 


6.33 


.12 


3.27 


3.27 


-.01 


5.82 


5.66 


.16 


382 


446 


476 


.6 


8.26 


8.02 


.24 


4.65 


4.45 


.21 


7.51 


7.28 


.24 


292 


342 


372 


1.0 


9.83 


9.53 


.30 


6.17 


5.62 


.56 


9.16 


9.03 


.13 


171 


198 


230 


1.4 


11.33 


10.44 


.89 


7.74 


6.32 


1.42 


10.92 


10.54 


.38 


81 


111 


97 


1.8 


12.68 


10.92 


1.76 


9.14 


6.74 


2.40 


12.71 


11.07 


1.65 


31 


35 


42 


2.2 


* 


* 


* 


10.00 


6.86 


3.14 


13.52 


11.81 


1.71 


0 


9 


21 


2.6 


* 


* 


* 


* 


* 


n 


14.75 


12.12 


2.63 


0 


0 


12 


3.0 


* 


* 


* 


* 


* 


* 


* 


* 


* 


0 


0 


0 
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Table 4 



NR, PNR and difference for a second spaced sample from the 
November 1982 administration* 







Section 


2 


Section 


3 


Section 
















2 


3 


TH 


nr 


pnr 


diff 


nr 


pnr 


diff 


N 


N 


-2.6 






* 


* 


* 


* 


0 


0 


-2.2 






* 


3.89 


3.01 


.88 


0 


9 


-1.8 








3.43 


3.08 


.35 


0 


14 


-1.4 


1.86 


1.64 


.22 


3.46 


3.2 


.26 


59 


41 


-1.0 


1.64 


1.80 


-.16 


3.65 


3.41 


.24 


124 


114 


-.6 


2.06 


2.02 


.04 


3.73 


3.82 


-.09 


216 


236 


-.2 


2.45 


2.54 


-.09 


4 .62 


4.54 


.08 


351 


315 


.2 


3.27 


3.31 


-.04 


5.63 


5.70 


-.07 


412 


449 


.6 


4.72 


4.44 


.28 


7.27 


7.33 


-.06 


358 


389 


1.0 


5.95 


5.79 


.16 


9.18 


9.06 


.12 


232 


252 


1.A 


7.79 


6.37 


1.42 


10.79 


10.69 


.09 


94 


103 


1.8 


9.13 


6.75 


2.38 


12.79 


10.99 


1.81 


31 


39 


2.2 


9.38 


7.16 


2.22 


13.81 


11.67 


2.14 


13 


26 


2.6 








14.38 


12.25 


2.14 


0 


13 


3.0 








* 


* 


* 


0 


0 
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Table 5 



Item parameter estimates » and Q Indices for section 
1 of the November 1892 administration 



Pos 


eta 


a 


b 


c 


Q 


1 


.18 


.53 


-2.68 


.16 


9.48 


2 


.33 


1.23 


-1.42 


.16 


11.55 


3 


.28 


1.04 


-1.94 


.16 


13.15 


4 


.25 


.99 


-1.27 


.16 


29.02 


5 


.29 


1.16 


-1.29 


.16 


11.88 


6 


.13 


.83 


-.82 


.16 


30.32 


7 


.26 


.79 


-1.06 


.16 


14.37 


8 


.26 


.77 


-.61 


.16 


65.32 


9 


.28 


1.48 


-.77 


.05 


10.05 


10 


.22 


.94 


-.76 


.06 


12.26 


11 


.11 


.77 


.46 


.16 


34.24 


12 


.23 


1.20 


-.65 


.11 


31.57 


13 


.18 


1.27 


-.22 


.10 


19.88 


14 


.15 


1.50 


.60 


.32 


28.29 


15 


.18 


.46 


-.34 


.16 


273.56 


16 


.18 


1.50 


-.07 


.32 


17.06 


17 


.19 


.97 


,09 


.04 


24.93 


18 


.09 


1.50 


-.13 


.06 


52.64 


19 


.12 


1.50 


.42 


.20 


14.34 


20 


.18 


1.50 


.54 


.11 


65.93 


21 


.28 


1.20 


-1.70 


.16 


30.96 


22 


.28 


.94 


-1.18 


.16 


12.97 


23 


.24 


.67 


-1.12 


.16 


23.54 


24 


.22 


1.32 


-.31 


.29 


10.47 


25 


.28 


.57 


-1.45 


.16 


48.23 


26 


.15 


.87 


-.77 


.16 


54.98 


27 


.22 


.88 


-1.53 


.16 


28.44 


28 


.25 


.87 


-.23 


.28 


11.60 


29 


.17 


1.29 


-.23 


.15 


10.57 


30 


.14 


.43 


-.18 


.16 


46.17 


31 


.12 


.61 


-.28 


.16 


25.94 


32 


.18 


1.00 


-. 08 


.27 


11.01 


33 


.28 


1.25 


-.14 


.18 


8.88 


34 


.18 


1.41 


.15 


.16 


17.38 


35 


.11 


1.50 


1.20 


.19 


44.75 


36 


.30 


.86 


-1.49 


.16 


25.75 


37 


.22 


.83 


.20 


.13 


8.20 


38 


.27 


.70 


-2.27 


.16 


6.75 


39 


.16 


1.03 


.00 


.20 


13.50 


40 


.24 


.74 


-.78 


.16 


7.04 


41 


.28 


1.18 


-.40 


.12 


15.22 


42 


.19 


.97 


-.62 


.23 


11.23 


43 


.26 


1.03 


-.87 


.16 


47.86 


44 


.24 


1.50 


-.44 


.30 


21.47 


45 


.22 


1.06 


-.88 


.31 


11.23 


46 


.14 


.95 


-.53 


.24 


4.66 


47 


.14 


.92 


.26 


.26 


18.96 


48 


.21 


1.18 


.16 


.26 


14.31 


49 


.13 


1.50 


-.01 


.38 


41.01 


50 


.20 


.88 


.33 


.38 


24.98 
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TABLE 6 



ITEM PARAMETER ESTIMATES, ETA AND Q INDICES FOR SECTION 
2 OF THE NOVEMBER 1892 ADMINISTRATION 



PCS 


ETA 


A 


B 


C 


Q 


1 


.24 


.60 


-2.41 


.19 


18.86 


2 


.33 


1.44 


-1.20 


.22 


94.S2 


3 


.33 


1.46 


-.87 


.37 


8.27 


4 


.33 


1.50 


-.80 


.22 


8.48 


5 


.26 


1.07 


-.61 


.19 


9.53 


6 


.26 


1.50 


-.48 


.23 


14.81 


7 


.16 


.69 


-.63 


.19 


33.62 


8 


.30 


1.47 


-.12 


.25 


13.11 


9 


.28 


1.50 


-.15 


.19 


7.42 


10 


.20 


1.08 


.01 


.13 


11.29' 


11 


.09 


1.29 


.31 


.10 


7.55 


12 


.19 


1.34 


.48 


.13 


13.88 


13 


.14 


1.35 


.47 


.08 


39.07 


14 


. 19 


1.13 


1.69 


.05 


34.44 


16 


.35 


1.04 


-.78 


.32 


12.51 


17 


.38 


1.07 


-1.36 


.19 


12.90 


18 


.30 


1.50 


-. 80 


.27 


11.85 


19 


,23 


1.14 


-.64 


.26 


13.32 


20 


.23 


1.39 


-.71 


.12 


21.99 


21 


.26 


1.50 


-.59 


.19 


10.40 


22 


.24 


1.03 


-.90 


.09 


8.92 


23 


.31 


.86 


-.55 


.14 


7.13 


24 


.19 


.90 


-.62 


.04 


10.35 


25 


.26 


.69 


-.64 


.19 


22.76 


26 


.20 


.92 


-.23 


.22 


46.15 


27 


29 








Q 7 1 


28 


.12 


1.50 


1.02 


.13 


47.40 


29 


.26 


1.06 


.02 


.29 


13.08 


30 


.23 


.48 


-.61 


.19 


20.00 


31 


. 16 


.91 


.13 


.29 


18.91 


32 


.22 


1.14 


-.14 


.15 


15.88 


33 


.32 


.73 


.05 


.18 


10.22 


34 


.14 


1.50 


.13 


.28 


14.04 


35 


.28 


.79 


.56 


.32 


12.77 


36 


.20 


1.17 


.64 


.25 


22.20 


37 


.13 


.92 


.43 


.05 


37.00 


38 


.15 


.80 


.89 


.20 


42.17 


39 


.03 


1.50 


.75 


.18 


27.36 
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Tablt 7 



Ifm paraMttr tttlMCtt» tta end Q indicts for ttctlon 
3 of the Novtabtr 2892 Adalnlttratlon 



Poi 


tta 


« 


b 


c 


Q 


I 


• 29 


U34 


-1.37 


• 24 


6.79 


2 


.21 


1.50 


-•77 


.30 


10. 69 


3 


.24 


1.49 


-.52 


.23 


8.78 


4 


.17 


1.50 


-.25 


.45 


16.81 


5 


.20 


.87 


-.10 


.<4 


21.29 


6 


.20 


.87 


-.63 


• 12 


11.94 


7 


.27 


• 72 


-.97 


.10 


33.94 


8 


.17 


.68 


-.84 


.10 


30.03 


9 


.28 


.92 


-.36 


• 26 


12.05 


10 


.23 


1.23 


-.33 


• 23 


5.55 


11 


.15 


1.25 


-.02 


.36 


13.76 


12 


.13 


.93 


.06 


.24 


6.41 


13 


.22 


1.18 


-•13 


.23 


18.16 


14 


.16 


.45 


• 35 


.10 


42.42 


IS 


.19 


1.13 


-.20 


.18 


14.10 


16 


.19 


.96 


-.31 


.11 


18.89 


17 


.14 


1.05 


.33 


.26 


9.06 


16 


.21 


1.06 


.22 


.26 


13.68 


19 


.15 


.86 


.15 


.13 


19.11 


20 


.14 


1.01 


-.12 


.05 


19.80 


21 


.23 


.35 


.05 


.10 


59.92 


22 


• 17 


1.50 


.41 


.25 


13.75 


23 


.20 


.76 


.06 


.16 


13.21 


24 


.14 


1.17 


.60 


.21 


36.49 


25 


.14 


• 99 


.62 


.23 


14.55 


26 


• 09 


1.10 


.32 


.13 


11.07 


27 


.03 


.96 


1.73 


.17 


13.19 


26 


• 12 


1.24 


1.20 


.29 


13.64 


29 


• 20 


1.45 


.49 


.17 


49.12 


30 


• 19 


1.15 


-.85 


.31 


9.62 


31 


• 22 


1.04 


-.39 


.17 


23.80 


32 


• 38 


1.50 


-. 86 


.06 


23.30 


33 


• 16 


.46 


.21 


.10 


61.59 


34 


• 31 


.95 


-1.61 


.10 


28.49 


35 


.27 


1.50 


-.61 


.10 


17.13 


36 


• 12 


1.07 


-.31 


.17 


21.29 


37 


• 30 


.73 


-» 66 


.10 


22.34 


36 


• 26 


.61 


-.56 


• 30 


44.96 


39 


• 26 


1.50 


-1.02 


.21 


9.86 


40 


• 21 


1.44 


.00 


.30 


33.90 


41 


• 25 


.61 


-.92 


.10 


8.14 


42 


• 19 


.62 


-2.51 


.10 


13.98 


43 


• 09 


1.01 


.40 


.11 


19.80 


44 


• 07 


1.19 


1.59 


.05 


11.71 


45 


• 19 


1.40 


.06 


.32 


14.79 


46 


• 12 


.64 


-.64 


.10 


22.26 


47 


• 14 


1.25 


.27 


.13 


15.66 


46 


• 21 


.65 


-1.43 


.10 


25.64 


49 


• 13 


.93 


1.79 


.17 


7.94 


50 


• 17 


1.29 


.26 


.22 


44.51 


51 


• 16 


.97 


.32 


.29 


8.45 


52 


• 17 


.50 


.47 


.10 


29.29 


53 


• 06 


1.10 


.65 


.15 


14.59 


54 


• 20 


1.30 


-.21 


.26 


15.25 


55 


• 27 


1.50 


-.16 


.10 


17.81 


56 


• 12 


.20 


.86 


.10 


14.02 


57 


• 15 


1.50 


-.16 


.24 


12.36 


56 


• 13 


1.50 


-.29 


.20 


24.72 
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Table 8 



Mean and standard deviations of TOEFL scores for 
the September, October, November, and December administrations 





September 




October 






November 




December 






Section 




Section 






Section 




Section 






1 2 3 


Total 


1 2 3 


Total 


1 


2 3 


Total 


1 2 3 


Total 


Mean 


51 49 50 


500 


52 49 49 


502 


53 


49 50 


507 


52 50 50 


508 


S.D. 


6.0 7.1 7.2 


60 


6.6 7.0 7.3 


63 


6.7 


7.3 6.9 


63 


6.6 7.2 6.9 


62 


N 


5332 




10793 






21359 




6167 




Form 


3FrF9 




3FrF10 






3FrFll 




3FrF12 
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Table 9 

Mean and standard deviations of item parameter on the 
September, October, November, and December 1983 
administrations on Section 2 



Standard 

Mean Deviation 





a 


b 


c 


a 


b 


c 


September 


J. 02 


-.24 


.19 


.33 


.93 


.07 


October 


1.02 


-.22 


.17 


.33 


.91 


.08 


November 


1.05 


-.32 


.16 


.33 


.88 


.09 


December 


1.03 


-.24 


.18 


.36 


.98 


.08 
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Figure 1 
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Figure 2 




# 50 
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Ability 



Fifura 3 

1 2 3 4 9 



Hardaat Itaaa 

31 32 33 36 39 40 



3.0 (highaat) 



TH» < 20th ptl) 

1 

2 

3 

4 

5 

6 

7 

8 

9 
10 



1 .2 .3 .9 1.0 

2 .3 .4 1.0 i.O 

3 .4 .5 .. 1.0 1.0 1.0 



9 1.0 1.0 1.0 1.0 1.0 
0 1.0 1.0 .. 1.0 1.0 1.0 
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Figure A 




Figure 5 




Figure 6 




Figure 7 
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Figure 9 
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Figure 10 
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Figure 12 
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Figure 13 
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Figure 15 
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Figure 16 
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Table Al 

Item paraneters estimates and Q foe section I 
of the September 1983 administration 



POS 


A 


B 


C 


Q 


1 


.88 


-1.51 


.08 


37.75 


2 


.46 


-1.90 


.08 


30.80 


3 


.89 


-1.27 


.08 


30.62 


4 


1.24 


-1.05 


.08 


20.50 


5 


1.10 


-1.06 


.14 


13.50 


6 


.78 


-.40 


.08 


65.89 


7 


.76 


-.84 


.08 


61.64 


8 


.99 


,17 


.13 


30.79 


9 


.79 


.80 


.24 


49.41 


10 


.81 


1.67 


.18 


36.63 


11 


.51 


-2.33 


.08 


8.01 


12 


1.26 


-1.31 


.18 


8.94 


13 


1.14 


-.26 


.06 


11.60 


14 


U02 


-.64 


.11 


21. 18 


15 


.89 


-.43 


. 17 


34.94 


16 


1.45 


-.38 


.22 


53.24 


17 


.46 


.47 


.08 


47.19 


18 


.92 


-.01 


.01 


85.45 


19 


.67 


-.98 


.08 


19.73 


20 


.87 


.65 


.26 


15.81 


21 


.40 


-.90 


.08 


19.34 


22 


1.15 


.65 


.23 


17.99 


23 


.90 


-.63 


.37 


11.92 


24 


I.IA 


-.98 


. 18 


20.02 


25 


1.20 


-1.47 


.08 


12.65 


26 


.95 


-.05 


.29 


7.74 


27 


1.29 


-1.01 


.12 


4.47 


28 


1.13 


-.84 


.07 


9.80 


29 


1.50 


-.52 


.13 


6.20 


30 


1.37 


.38 


.19 


22.81 


31 


.36 


.03 


.08 


22.64 


32 


1. 12 


-.83 


.11 


8.50 


33 


.94 


.70 


.29 


6.18 


34 


1.03 


.29 


.14 


10.64 


35 


1.50 


-1. 16 


.13 


23.19 


36 


.85 


-1.17 


.08 


7.78 


37 


1.29 


-.62 


.12 


8.49 


38 


1.13 


-.45 


.14 


4.26 


39 


1.50 


-.35 


.28 


5.36 


40 


1,17 


-.21 


.21 


6.94 


41 


1.50 


.31 


.11 


13.75 


42 


.46 


-1.45 


.08 


5.66 


43 


.47 


-1.50 


.08 


7.49 


44 


.89 


-1.58 


.08 


7.39 


45 


- 27 


- 21 


. uo 




46 


.94 


1.05 


.16 


20.21 


47 


1.07 


-.26 


.09 


12.45 


48 


.56 


.71 


. 19 


15.49 


49 


.89 


-.94 


.05 


29.47 


50 


1.13 


.34 


.36 


7.64 



51 



Table A2 

Item parameters estimates and Q for section 2 
of the September 1983 administration 



POS 


A 


B 


C 


Q 


1 

1 


• o2 


1 £.n 

-1.62 


. 17 


16. 50 


•> 




1 o 

18 


. 17 


21. 76 


J 


1 AO 


A 1 

91 


. 17 


8. 89 


A 
*♦ 


• 9o 


46 


.21 


17.40 


c 


1 


49 


.03 


31. 75 


0 


1 

1* 


O 1 

. Jl 


. 32 


26. 18 


7 




^o 
. oZ 


. 17 


41. i8 


a 
O 


1* UU 


1 AO 

1 . uz 


AO 

. Oo 


17.86 


Q 


1 10 

1 • JZ 




o c 
. Zo 


1 7 O O 

1 /. Z2 


lU 


1 Aft 


— ft 1 


1 ^ 


OA 0 O 

Z4. jy 


1 1 
1 1 


7A 
• /O 


— ^ 1 


\ 7 


OA OA 
ZU. Z4 




SA 


— 


1 7 
. 1 / 


1ft 0 1 

lo. Zl 






— ^ft 
— . JO 


1 7 
. 1 / 


AO 1 A 

'♦y. 10 


1 A 


07 


.Hi 


10 


1 1 Q7 
11.0/ 


1 ^ 


• 




1 7 
. 1 / 


0 Q O O 


1 A 
ID 


1 no 


— 07 
— . Z/ 


OA 

. zu 


J. JJ 


1/ 


1 1 7 


OA 


1 o 

. lo 


IJ. 94 


1 ft 


1 

1 • DV 


0 1 A 

Z. lU 


0 0 

. ZZ 


40. 03 




1 • ZJ 


AQ 

. Uo 


0 A 


"7 If. 

7. 74 






— 1.1/ 


. 1/ 


2Z. 5o 




1 OA 
1 • ZO 


1 A 
— . 1^ 


1 O 
. 1 J 


24. 36 




• 00 


O 1 A 

-Z. 14 


• 17 


1 A rv o 

10. 98 


ZJ 


1 *3 C 

1 • 33 


1 OA 


1 o 

. 12 


15. 85 


0 A 


1 no 
1 • Uj 


1 "7 C 


. 1/ 


O /I 

o. 41 




1 0£. 

1* Zo 


-. 26 


. 13 


21.35 


26 


1.15 


-.76 


.19 


15.51 


27 


.89 


l.Ol 


.19 


12.98 


28 


.66 


-1.60 


.17 


13.96 


29 


.93 


-1.08 


.17 


13.33 


30 


.35 


-.15 


.17 


25.34 


31 


1.25 


-.64 


.24 


15.86 


32 


1.50 


-.64 


.18 


12.20 


33 


.75 


.13 


.15 


24.71 


34 


1.38 


-.47 


.10 


11.32 


35 


1.19 


-.18 


.31 


7.53 


36 


1.50 


.73 


.10 


14.37 


37 


.97 


1.05 


.14 


27.28 


38 


.97 


.93 


.20 


5.01 
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TAblt A3 

Itta p«rAMttrf ••tiMt«f And Q for ttction 3 
of tht S«pttBb«r 1M3 adAlniitration 



POS 


A 


B 


C 


q 


1 


.40 


-2.30 


.25 


16.66 


2 


.47 


-2.76 


.23 


6.76 


3 


1.50 


-1.06 


.14 


26.04 


4 


' 1.23 


-1.22 


.10 


16.41 


5 


.36 


-2.19 


.23 


27.78 


6 


1.38 


-.52 


.23 


28.24 


7 


.78 


-.46 


.25 


65.00 


8 


1,50 


.13 


.34 


30.44 


9 


1.29 


.02 


.19 


29.03 


10 


1.10 


.41 


.27 


31.30 


11 


1.50 


1.22 


.21 


23.43 


12 


.99 


.89 


.07 


21.81 


13 


.92 


1.05 


.33 


68.81 


14 


.89 


1.01 


.18 


40.33 


15 


1.30 


1.33 


.28 


24.28 


16 


• 83 


-.13 


.47 


27.39 


17 


.91 


-.48 


.50 


64.92 


18 


1.50 


-.33 


.40 


27.28 


19 


.31 


.33 


.23 


34.33 


20 


1.36 


.60 


.08 


22.00 


21 


.74 


.02 


.10 


42.26 


22 


1.49 


-.34 


.06 


24.63 


23 


1.03 


-.90 


.06 


31.12 


24 


.86 


-.47 


.10 


30.12 


25 


1.30 


.40 


.09 


37.66 


26 


1.30 


-.01 


.18 


49.61 


27 


1.47 


-.22 


.11 


67.39 


28 


1.17 


-.33 


.03 


60.18 


29 


1.30 


.34 


*27 


13«70 


30 


.79 


.11 


.09 


10.31 


31 


.91 


-1.36 


.23 


19.63 


32 


.63 


-. 33 


.23 


7.33 


33 


1.08 


-.87 


.19 


4.34 


34 


.84 


-1.03 


.06 


7.74 


35 


.96 


-.43 


.12 


11.93 


36 


1.38 


-.28 


.13 


20.03 


37 


.41 


.06 


.23 


12.18 


38 


.73 


-.34 


.23 


11.72 


39 


1.13 


.63 


.18 


17.61 


40 


1.11 


1.17 


.37 


13.18 


41 


.84 


.70 


.23 


10. 24 


42 


1.30 


h78 


.30 


9.12 


43 


.96 


.73 


.12 


21.64 


44 


1.08 


1.00 


.20 


14.49 


45 


.80 


.07 


.13 


12.60 


46 


.33 


-.88 


.23 


26.02 


47 


.11 


.17 


.20 


11.07 


48 


.98 


.36 


.18 


19.33 


&9 


1.11 


-1.03 


.21 


10.37 


50 


.80 


-1.20 


.23 


33.11 


51 


.71 


1.02 


.16 


40.08 


52 


.37 


-1.84 


.23 


17.83 


53 


.68 


.91 


.13 


11/91 


54 


1.23 


.03 


.11 


13.63 


55 


.91 


-.70 


.12 


32.37 


56 


1.37 


-.33 


.19 


8.31 


57 


1.12 


-.21 


.22 


26.94 


56 


.63 


.90 


.16 


16.66 
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Table A4 

Item parameters estimates and Q for section 1 
of the October 1983 administation 



POS 


A 


B 


c 


0 


1 


.54 


-2.47 


.08 


8. 82 


2 


1.03 


-1.40 


.08 


32. 08 


3 


• 86 


-1.36 


.08 


15. 16 


4 


.87 


-.13 


.24 


5.68 


5 


.78 


-1. 10 


.08 


19* 57 


6 


.98 


-1.03 


.23 


11.05 


7 


1.22 


.41 


. 15 


24.27 


8 


1.28 


2.07 


. 19 


10.08 


9 


.53 


.00 


.08 


17.33 


10 


.68 


-2.02 


.08 


20.95 


11 


.99 


-.78 


. 16 


21.84 


12 


.55 


-. 16 


.08 


20. 00 


13 


.61 


-.66 


.08 


12.19 


14 


.61 


.27 


.0 


44.24 


15 


.75 


-.96 


.08 


27.81 


16 


.91 


.44 


.24 


22.46 


17 


.95 


.55 


.22 


29.64 


18 


1.19 


-1.21 


.22 


11.30 


19 


.48 


• 25 


.08 


10. 18 


20 


.96 


.79 


.42 


71.95 


21 


1.08 


-.04 


.27 


20.03 


22 


1.37 


-.51 


.35 


17. 21 


23 


.85 


-.62 


.33 


41. 84 


24 


.77 


1.44 




3K70 


25 


1.09 


-. 10 


.37 


9.50 


26 


.91 


-1.42 


.08 


11.57 


27 


.84 


-1.74 


.08 


27.45 


28 


.99 


-.76 


.08 


5. 76 


29 


1. 13 


-.62 


. 17 


4.53 


30 


.71 


-.87 


.08 


11.44 


31 


.77 


-. 18 


.23 


10. 79 


32 


.93 


-.09 


.09 


14. 17 


33 


1*32 


. 12 


.20 


8.48 


34 


.49 


-.98 


.08 


21.47 


35 


.31 


-1.42 


.08 


24. 57 


36 


1.50 


.81 


. 15 


13. 90 


37 


1.06 


-1.00 


. 13 


22.45 


38 


1.49 


.05 


.30 


17.23 


39 


.69 


-1.33 


.08 


6.07 


40 


.67 


-.82 


.08 


16.75 


41 


1.50 


.23 


.25 


12.23 


42 


1.50 


.13 


.05 


12.84 


43 


1.43 


.08 


.14 


20.06 


^^ 


1. 50 


-.42 


. 11 


20.61 


45 


.46 


-.93 


.08 


28.42 


46 


1.03 


-.13 


.12 


24.50 


47 


1.29 


-.91 


.11 


23.25 


48 


1.19 


-.06 


.11 


53.72 


49 


1.23 


-.79 


.15 


27.30 


50 


1.25 


-.35 


.26 


4.77 
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Table A5 

Item parameters estimates and Q for section 2 
of the October 1983 administration 



ruo 


A 

A 


i> 






1 
1 


ftft 


1 • OO 


1 A 
.14 


9 9 ftA 
ZZ. 04 


0 

im 


79 


• 


no 


10 9 1 
1 7. Zl 




1 LC\ 

1 • 


- 7A 


1 'I 
. 1 J 


ft HQ 

0. U7 


A 


1 A9 


^1 1 1 
• 1 1 




in 1 Q 
lU. 1 J 


<^ 


1 1 1 


— A 1 


. JO 


1 ^ HA 
1 J. U4 
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V 


1 sn 


1 nn 
1 • uu 


1 A 
. 14 


9 1 ftA 
Z 1 . OD 


7 
» 


ftQ 

• 07 


AQ 

• 07 


. 1 J 


9n 
zu. 


Q 
O 






1 A 
. lU 


9 Q C9 
Z J. 


Q 




^1 no 

"^1 • U7 


1 A 
.14 


A 7ft 
D. /O 


in 


1 "19 


— 99 


. ZD 


A no 
0. Uo 


1 1 




"1 • OU 


. 14 


1 J. 00 


1 *) 


ft7 
• 0/ 


— ftA 
o4 


1 A 
.14 


1 A OQ 
14. UJ 




• JO 


^1 on 

"1 • 7U 


1 A 
. 14 


1 1 Aft 
11 . Do 




1 


~» Ou 


9 A 
. ZO 


7ft TX 
/O. / J 


1 > 


fi9 


— ^ft 
• jO 


1 A 
. 14 


9n on 

zu. 7U 


If) 


Q 1 
• 7 1 


— 1 A 
• ID 


99 
. ZZ 


A Oft 
D. 70 




• 0 1 


1 1 ^ 


9ll 

. zu 


ZJ. /u 


18 


• oo 


fi9 


. U J 


Q^ AA 
7 J. D4 


1 Q 


in 


1 ft7 
1*0/ 




9 1 A'^ 
Z 1 . D J 


20 


• Ov 


- AQ 
• 7 


1 9 
. 1 z 


9Q Q 1 

Z 7. 7 1 


21 


1. • V 7 


1 • 7'* 


.14 


9*^ L 1 

^ J . H 1 




RR 
• oo 


- AQ 

• H 7 


HQ 

. U7 


ft 7Q 

0. / 7 




1 1 n 


— 1 A 


1 7 
. 1 / 


in ft 1 

lU. 01 




1 Aft 

1 • 40 


9 A 

• zo 


1 ft 
. 10 


1 A no 

14. Kjj 




1 

1 • j\) 


1 'I'l 
i* J J 


• U/ 


9ft nn 
Zo* UU 




1 • J/ 




9 1 
. Zl 


Q A 9 
7. 4Z 


97 


1 A 1 




9 A 
. Z4 


A Q9 
D. OZ 


9ft 


Aft 
• '♦o 




1 A 
. 14 


ZD. j9 


90 


1 QH 




1 C 

. Id 


1 C O 0 
1 J. OJ 




1 n<; 
1 • 


_ Art 


9 7 

. z/ 


OA OA 

ZU. 04 


31 


.76 


-.47 


.16 


7.90 


32 


.99 


-.28 


.22 


9.64 


33 


1.01 


-1.08 


.07 


12.84 


34 


.89 


-.27 


.15 


11.66 


35 


1.21 


.03 


.16 


27.39 


36 


1.09 


1.47 


.n 


17.30 


37 


1.01 


.63 


.29 


11.13 


38 


.76 


.45 


.29 


19.94 
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Table A6 

Item pArAMtert ettlMtet And Q for ttction 3 
of the October 1983 AdalnittrAtion 





1 


1 

m 


Vt 


n 
X 


1 


1 7A 
1 • 20 


mm\ 77 


1 A 
• 15 


O 79 
7* d2 


* 


• dO 


■"1 • /o 


1 7 


A AA 
0^ 0«l 




i • ZO 


• 7* 


77 
• d J 


1 0 77 
1 ?• dd 


A 
•# 


1 nR 


— R9 
• OZ 


90 
• ^7 


OO 7 1 

22^ dl 




• V 1 
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• 


OA no 

2*»^ U7 


w 


1 AO 


• Uw 


97 


in OA 

lU^ 71 


7 




— HI 
• Oi 


• i ^ 


I 0 70 
1 ?• d7 


8 


1 35 


\ 7 


91 


1 7 SA 

1 ' • 50 


Q 


77 


• 00 


1 0 

• 17 


10 7A 
1 ?• /O 


10 


7Q 

• / 7 


• Od 


9A 
• 20 


17 A 1 
1 / • Hi 




• dO 


1 70 
1 • J7 


99 
• 22 


OO A7 
22^ Hd 




1 OA 


07 
• 7d 


O 1 

• 21 


OC O 1 

26^ 21 


1 7 


1 'y^ 
!• 2d 


on 
• oU 


O A 

• 2h 


L. 1 O 

6^ lo 


1 A 


1 AO 
!• «*7 




• 20 


22^38 


1 c 


• /o 


• 23 


• 30 


9^58 


1 A 
10 


1 

!• dU 


"• 7*» 


• A7 


12^58 


1 7 
1 / 


1 7 A 


AC 

"-•*»0 


L L. 

• A6 


2U28 




• 3A 


• Al 


• 13 


26^ 97 


1 Q 


• do 


• ^0 


• 11 


12^32 


*it\ 

2U 


• ol 


1 nil 


• A6 


30^99 


21 


• 70 


• 60 


• 19 


13^37 


22 


1 1 u 

1* io 


nn 


OO 

• 29 


1 U67 


2 J 


1 on 
!• 2U 


— *>n 
-• 2U 


• 28 


\ L. CO 

16^ 6d 


2*» 


1 

!• dU 


_ no 


1 o 
• l7 


lA^ol 


2d 


1 

!• dU 


1 A 


1 n 
• lU 


27^00 


7A 
aO 


^7 
• dJ 


. A7 


1 7 

• Id 


o O A 1 

23^ Al 


2/ 


1 Ar^ 


20 


o / 

• 2A 


20^81 


2o 


1 cn 
l«dO 


-•OA 


• 26 


8^ 58 


27 


O Q 

• 20 


cn 
• dU 


1 o 

• 13 


12^35 


JU 


1 cn 


OO 

-• 33 


on 

• 39 


8^ 62 


1 1 


• oo 


-• 00 


• 18 


32^ 20 


J2 


• do 


OA 


1 o 

• 13 


8^23 




U 05 


-• 78 


• 2A 


3^2A 


7 A 


!• AA 


-• 73 


• 33 


2.52 


7^ 
Jd 


• oO 


-• 82 


• 13 


B* AS 


7A 


on 
• oU 


-• 25 


• 16 


13^36 


77 


1 71 


-• lA 


• lA 


39^ A 1 


7U 


TO 
• 77 


"• d/ 


• 09 


28^23 


70 
J7 


1 10 

!• 1 J 


*> A 

• 2A 


o/\ 

• 30 


17^ 51 


An 


• Od 


-•06 


• 22 


3U 18 


A 1 
•♦1 


Q A 


io 
• 62 


• 25 


lA^ 51 


•♦2 


1 lO 

!• 23 


• 53 


• 23 


10^88 


A 7 


• 00 


• 2A 


• 06 


20^70 


A A 


1 oo 

!• J7 


OO 

• 33 


• 21 


6^05 


A^ 


1 cn 
!• dU 


!• 66 


• 35 


9^87 


AA 
•♦0 


1 n£. 
!• UO 


1 iO 

— 1 • 6J 


• 13 


6^05 


A7 


AR 
• OO 


— 1 77 


1 7 
• id 


lo^o/ 


AS 


1.27 


• 06 


• lA 


6^A5 


A9 


K07 


-•22 


.22 


16^7A 


50 


• AO 


-•3A 


• 13 


13^22 


51 


K30 


-•00 


• 17 


7^88 


52 


K03 


• A5 


• 25 


10.38 


53 


• 67 


• 79 


• 20 


11.68 


5A 


U20 


*70 


• AO 


17.15 


55 


K30 


• 71 


.27 


8.87 


56 


U50 


• 32 


• 32 


13.03 


57 


K50 


• 22 


• 31 


22.76 


58 


K29 


• AO 


• 26 


5.19 
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Table A7 

Item parameters estimates and Q for section 1 
of the November 1983 administration 



POS 


A 


B 


c 


Q 


1 

X 


1 o c 
!• Z^ 


1 An 

-1 • 40 


.34 


8. 82 


I 


1 AC 

1.05 


1 1 / 
-1. 14 


.43 


7,40 


3 


# 90 


-1.32 


1 ft 
. 19 


8. 06 


/, 
*f 


1 n"? 
It 0/ 


1 1 A 

-1, 14 


• 27 


1 O P ft 

12. 50 


5 


ft c 

.96 


38 


.01 


129. 04 


0 


o c 


7ft 

-. 79 


1 ft 
. 19 


36.49 


7 

1 


ft c 

. 95 


33 


• 00 


80. 35 


o 
0 


1 1ft 
1. 19 


. 26 


. 11 


11 O 1 

11.31 




1 cn 


77 


. 27 


32. U5 


1 n 


1 AO 

1 • 49 


A O 

. 48 


1 o 

. 18 


"7 £ CO 

76. 52 


1 1 
11 


Q A 

• 94 


1 A l'^ 

-1 . 40 


1 ft 
. 19 


1 O "7 A 

13. 74 


1 9 
l£ 


• 00 


_ Aft 


1 Q 

. 19 


Q/. 7Q 

o^. /o 


1 J 


1 OA 


n7 


O A 
. Z4 


0 0 cn 


1 A 
1*1 


Q/i 


1 7Q 


1 Q 

. 19 


11 A 1 
J J. 0 J 


1 ^ 


A1 




1 Q 

. 19 


AO AO 
09. 09 


1 A 


7 ^ 


AA 


1 7 
• if 


0 C Q C 

Z^. o^ 


1 7 


AO 


Q 1 


nc 


O Q Q/. 
Z J. 


1 

lo 


1 0 7 
1 « Z/ 


Qn 


1 0 

. IZ 


o ii 1 A 

Zo. lo 


1 Q 
l7 


o c 


1 Q 

19 


1 ft 
. 19 


CO n£. 

53. 06 


on 


1 Oft 

1. 29 


7n 

70 


• 16 


12.09 


O 1 

21 


1 O A 

1. 


A 1 

-.41 


1 A 

. 14 


1 / "7 C 

14. 75 


22 


• 47 


O TT 

-2. 77 


1 ft 
. 19 


20. 90 


23 


1.09 


.51 


.29 


6. 78 




.69 


. J5 


. 28 


26. 56 


25 


.86 


-1. 13 


• 0 


5. 55 


26 


ft /\ 
. 90 


-1. 10 


. 19 


7, 70 


27 


1.24 


81 


. 17 


14.63 


28 


1 1ft 
1. 19 


20 


. 33 


12« 12 


29 


ft o 

. 98 


1 CO 

-1. 52 


1 ft 
. 19 


O "7 O £ 

27. 26 




Q A 
• 9'» 


1 o 
IZ 


1 c 

. 15 


1 C ') 1 

15. Zl 


1 1 
Jl 


• by 


A O 

4Z 


1 ft 
. 19 


1ft Aft 

19. 40 




1 cn 


A Q 
. ^9 


1 D 

. 18 


1 C 7A 

15. /o 


J J 


1 0 1 


_ 1 7 
— . 1 / 


If) 
. lU 


11 /. Q 

1 1 • '♦y 


1A 


1 0 Q 


_ f\/. 

— . U** 


1 Q 

. lo 


A 1 1 
0. 1 1 


1 Q 


77 


1 O 0 

"1 . 8Z 


1 ft 

. 19 


1 O A A 

18. 44 


JO 


1 o c 

1. 25 


1 o o 

-1 . 22 


. 17 


1 /\ ft 1 
10. 91 


IT 

37 


1 A /\ 

1. 40 


38 


. 16 


10. 42 


38 


1. 24 


65 


. 17 


12.41 


39 


1 o o 

1. 32 


20 


. 12 


12. 57 


40 


.92 


. 81 


. 12 


18.09 


41 


1* 30 


-.06 


. 14 


13. 55 


A n 

42 


.57 


-. 08 


. 19 


o c o / 

25. 34 


A O 

43 


. 67 


1 /* n 

-1. 60 


1 ft 
. 19 


20. 19 


A A 


. 87 


1 1 /\ 
-1. 10 


1 ft 
. 19 


1 "7 Oft 

17. 29 


45 


.91 


-1.37 


.19 


12.23 


46 


1.13 


-.57 


.0 


3.32 


47 


.84 


-1.29 


.19 


16.08 


48 


.47 


-.87 


. 19 


14.75 


49 


1.38 


.33 


.13 


12.41 


50 


1.12 


-.65 


.14 


17.28 
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Table A8 

Item parameters estimates and Q for section 2 
of the November 1983 administration 



POS 


A 


B 


C 


Q 


1 


.82 


-1.87 


.14 


10.27 


2 


.60 


-.21 


.14 


89.37 


3 


.69 


-1.45 


.14 


7.50 


4 


1.49 


-.37 


.12 


11.32 


5 


1.15 


-.71 


.01 


12.41 


6 


1.17 


.23 


.09 


18.44 


7 


.47 


.91 


.14 


83.40 


8 


1.47 


.72 


.05 


33.27 


9 


1.27 


-1.09 


.05 


13.62 


10 


.91 


-.73 


.32 


20.56 


11 


.87 


-1.05 


.14 


30.12 


12 


1.36 


-1.40 


.36 


12.35 


13 


.85 


-.08 


.34 


21. 18 


14 


.95 


-.51 


.10 


8.36 


15 


.67 


-. 93 


.14 


30.58 


16 


1.20 


1.19 


.19 


9.91 


17 


.59 


-.32 


.14 


17.39 


18 


1.29 


-.20 


.03 


10.94 


19 


1.40 


-.35 


.07 


22.62 


20 


1.50 


.81 


.37 


47.41 


21 


1.50 


1.40 


.22 


26.94 


22 


1.10 


-1.57 


. 14 


9.98 


23 


.64 


-2.26 


. 14 


11.04 


24 


1.04 


1.10 


.27 


18.46 


25 


1.50 


.25 


.19 


14.77 


26 


1.12 


-.45 


.11 


16.69 


27 


1.50 


.90 


.18 


27.21 


28 


.69 


-.92 


.14 


14.57 


29 


1.33 


-.92 


.10 


7.74 


30 


1.05 


-1.27 


.14 


6.41 


J 1 


1 on 


-. 52 


.09 


19.29 


32 


1.37 


-.48 


.17 


13.56 


33 


1.02 


-.41 


.10 


6.28 


34 


.69 


-.03 


.08 


10.14 


35 


.77 


-.57 


.0 


15.95 


36 


.74 


.56 


.25 


12.61 


37 


.58 


-.09 


.14 


10.49 


38 


1.50 


.48 


.32 


15.04 
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Ttblt A9 

It«» pArtMttri titlMCM And Q for itction 3 
of the Nove»b«r 1983 «d»inlitr4tlon 



POS 


A 


1 


C 


q 


1 


• 94 


-2,01 


.11 


31.41 


2 


U23 


-.96 


.32 


39.82 


3 


1.13 


-. 98 


.20 


90.23 


4 


.82 


-.42 


.21 


21.85 


5 


1.04 


-.52 


.0 


12.60 


6 


.36 


-. 55 


.11 


18.19 


7 


.81 


.06 


.31 


21.49 


8 


.74 


.54 


.06 


16.75 


9 


.74 


-.33 


.11 


20.78 


10 


.48 


-.39 


.11 


24.28 


11 


1.50 


.29 


.10 


11.13 


12 


.73 


.43 


.15 


13.17 


13 


1.06 


.93 


.32 


34.48 


U 


1.41 


1.33 


.12 


10.90 


13 


1.33 


.78 


.21 


15.24 


16 


1.30 


-.16 


.29 


14.98 


17 


1.09 


69 


.28 


11.12 


18 


1.06 


-.24 


.14 


5.94 


19 


.84 


70 


.11 


39.06 


20 


.66 


.15 


.26 


13.17 


21 


1.07 


-.35 


.35 


5.81 


22 


.87 


.30 


.32 


7.13 


23 


1.50 


.28 


.27 


5.11 


24 


1.44 


-.05 


.18 


11.30 


25 


1.23 


-.46 


.22 


41.68 


26 


1.50 


-.08 


.12 


3.19 


27 


1.39 


.47 


.23 


9.46 


28 


1.23 


.35 


.07 


36.47 


29 


1.15 


-.03 


.25 


8.93 


30 


1.31 


-.82 


.24 


13.45 


31 


.65 


-2.14 


.11 


32.24 


32 


1.41 


-.79 


.20 


13.88 


33 


1.30 


-.22 


.25 


9.19 


34 


1.50 


-.18 


.11 


14.95 


35 


.62 


-.57 


.11 


8.35 


36 


.51 


-.18 


.11 


27.98 


37 


.58 


53 


.11 


19.99 


38 


.76 


.29 


.20 


12.07 


39 


.98 


.47 


.17 


13.03 


40 


.61 


.21 


.04 


41.58 


41 


1.03 


.02 


.20 


10.19 


42 


.75 


.71 


.27 


8.03 


43 
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